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Smart SPHERES is a prototype free-flying space robot based on the SPHERES platform. 
Smart SPHERES can be remotely operated by astronauts inside a spacecraft, or by mission 
controllers on the ground. We developed Smart SPHERES to perform a variety of intrave- 
hicular activities (IVA), such as operations inside the International Space Station (ISS). 
These IVA tasks include environmental monitoring surveys (radiation, sound levels, etc.), 
inventory, and mobile camera work. In this paper, we first discuss the motivation for free- 
flying space robots. We then describe the development of the Smart SPHERES prototype, 
including avionics, software, and data communications. Finally, we present results of initial 
flight tests on-board the ISS. 


I. Introduction 

F UTURE human space missions in Earth orbit, to the Moon, and to distant destinations offer many new opportu- 
nities for exploration 6 . However, astronaut time will always be in short supply, consumables (e.g., oxygen) will 
always be limited, and some work will not be feasible, or productive, for astronauts to do manually. Remotely oper- 
ated robots, however, can complement astronauts by performing this work under remote supervision by humans 
from a space station, spacecraft, habitat, or even from Earth. 

Telerobots, particularly semi-autonomous systems, can increase the performance and productivity of human 
space exploration. Telerobots are well suited to performing tasks that are tedious, highly repetitive, dangerous or 
long-duration, such as routine in-flight maintenance, systematic surveys, etc. Telerobots can also provide side-by- 
side assistance to astronauts, during both intravehicular and extravehicular activities (IVA and EVA). Telerobots can 
also perform “follow-up” work to complete or supplement tasks started by humans 1,5 . 

As part of NASA’s “Human Exploration Telerobotics” (HET) project 5 , we are currently developing a free-flying 
space robot called “Smart SPHERES”, which we describe below. Several free-flying robots have previously been 
proposed for a variety of mobile IVA tasks on-board human spacecraft 1 ' 3 ' 7 . These tasks include spacecraft health- 
management, environmental monitoring surveys (air quality, radiation, lighting, sound levels, etc.), automated logis- 
tics management (inventory, inspection, etc.), and crew activity support (mobile camera, automated documentation, 
interactive procedure display, etc). 

Although there have been previously free-flying robot demonstrations on ground and in space, none has focused 
on ground control of an IVA free-flyer in space. For example, theory and scientific principles for ground control of 
an IVA free-flyer were formulated in 1999 by the “Personal Satellite Assistant” (PSA) project 3 , but no flight demon- 
stration was ever performed. Other free-flyers have been tested in space, such as the “Autonomous Extravehicular 
activity Robotic Camera” (AERCam) Sprint deployed on the Space Shuttle STS-87 flight, but were only remotely 
operated from a short distance 11 . 


* Intelligent Robotics Group, NASA ARC, M/S 269-3, Moffett Field, CA 94035. AIAA Associate Fellow. 
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II. Use Cases for a Telerobotic IVA Free-Flyer 

Free-flying space robots can be used when humans are present to off-load routine work, to increase crew produc- 
tivity, and to handle contingencies. The International Space Station (ISS), for example, is a continuously manned 
orbital laboratory the size of a large house, which contains many thousands of inventory items and hundreds of di- 
verse payloads and experiments - all of which have to be managed by 3 to 6 person crew. 

Free-flying space robots can also be used during quiescent periods (i.e., when spacecraft are unmanned) to per- 
form spacecraft caretaking. As humans prepare to venture deeper into space, consideration is being given to devel- 
oping an intermittently manned “waypoint” facility, which would serve as a gateway to cis-lunar space and beyond. 
A free-flying robot would allow monitoring, maintenance, and repair of such a facility before, and between, crews. 

A. Spacecraft Health Management 

Currently ISS crew is required to perform periodic interior environmental surveys including air sampling and 
sound surveys, which require approximately eight hours of crew time per month. An air quality sensor equivalent to 
the current CSA/CSACP (Compound Specific Analyzer/ Compound Specific Analyzer Combustion Products) in- 
strument used on the ISS by the crew could be incorporated into the free-flyer and the ground could perform the 
periodic air quality surveys during crew sleep periods. Similarly, a microphone in the IVA free-flyer could be used 
by the ground to conduct the periodic sound surveys. Radiation and illumination surveys could also be performed by 
ground operators if the free-flyer was equipped with appropriate sensors. 

Contingency scenarios could also be supported by an IVA free-flyer. When smoke or a suspected fire is detected 
on the ISS, the current plan calls for the crew to retreat into the Soyuz spacecraft and shut the hatch until the situa- 
tion is analyzed by the ground. A free-roaming IVA free-flyer with an air quality sensor could significantly assist the 
ground in assessing the environment within the ISS, expediting the recovery from the event, or qualifying the criti- 
cality of a fire event. Similarly, in the unlikely event of a de-manning of the ISS due to possible problems such as 
issues with resupply vehicles, an IVA free-flyer could be used by the ground to assess the environment of the ISS, 
supporting analysis and prepartion leading to the re -manning of the station. 

B. Automated Logistics Management 

On-board logistics management includes keeping track of all the inventory of the ISS. In addition to dedicated 
or planned storage locations for items brought to the ISS by the resupply vehicles and stored by the crew, RFID (ra- 
dio frequency identification) is being incorporated in some items to assist the crew in locating them. It is expected 
that an IVA free-flyer would incorporate an RFID sensor, and that periodic (or continuous) RFID-based inventory 
surveys would be conducted to help keep track of the location of items. In particular, this capability could assist the 
crew in searching for things prior to an IVA activity. 

C. Crew Activity Support 

There are several areas of crew activity support that could be provided by an IVA free-flyer. The crew is often 
called upon to check if things are connected, look at equipment and payload displays, etc, and a free-flying camera 
could off-load these kind of tasks by providing ground controlled imagery of the area in question. Also, camera 
setup for activities onboard are frequent and time consuming, plus there is limited camera availability - a fly-around 
camera could mitigate some of this difficulty in obtaining camera coverage. Approximately two hours per month 
crew-time could be saved by this type of imagery support. Monthly ISS safety video surveys are required to be per- 
formed by the crew, and a free-flyer would be able to perform this routine task, which currently takes the crew ap- 
proximately one hour per month. 

A mobile video device could be used to "look over the shoulder” of the crew while performing an IFM (In Flight 
Maintenance) activity, giving the ground operators more information in real-time to support the activity. Crew pro- 
cedures could also be augmented by a free-flyer, for the crew to record comments in real-time, or for the free-flyer 
to “read” the procedures to the crew if the device is equipped with spoken language capability. With the addition of 
a display screen, the free-flyer could offer hands-free procedure image support, showing diagrams, pictures, and 
other relevant graphical information. 

Futhermore, ground control would gain the ability to have two-way visual interaction with the crew. For exam- 
ple, if there were uncertainty about the next step of a procedure to perform, due to actual conditions differing from 
what was expected, ground control could use an image captured by the free-flyer, circle the part, and have it dis- 
played to the crew member. This would be a far more precise and error-free means of resolving confusion than is 
currently possible using only voice communications. 
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III. SPHERES 


The Synchronized Position Hold Engage and Reorient Experimental Satellites (SPHERES) are volleyball-sized 
free-flyers that were developed by the Massachusetts Institute of Technology as a platform for testing spacecraft 
guidance, navigation, and control algorithms in micro-gravity 8 . Three SPHERES were deployed to the International 
Space Station (ISS) in 2006 and have been operated by NASA as an ISS facility since 2010. To date, astronauts 
have conducted more than 50 test sessions to study formation flying, rendezvous, and docking, as well as to perform 
on-orbit student competitions. 



Figure 1. Three SPHERES free-flyers have been on the ISS since 2006 (NASA ISS016-E-014220). 

Each SPHERES unit (Figure 1) is fully self-contained with propulsion, power, computing, and navigation 
equipment. A cold-gas (carbon dioxide) thruster system is used for motion control and DC batteries provide elec- 
tronic power. For processing, the SPHERES rely on a digital signal processor, which handles all on-board software 
functions including navigation, device control, and communications. An external ultrasonic, local positioning sys- 
tem provides data to estimate the position and orientation of SPHERES. The SPHERES’s original expansion port, 
which contains data, electrical, and mechanical interfaces, was upgraded in 2011. The expansion port allows exter- 
nal payloads to be attached to, and to control SPHERES. 

IV. From SP H ERES to Smart SPHERES 


A. Smartphone upgrade 

Because the SPHERES were originally designed as a satellite testbed for studying spacecraft control algorithms, 
they require modification in order to function as telerobots. In particular, the original SPHERES lack a high- 
performance, general-purpose processor for running modern robotics software. In addition, the SPHERES do not 
have the sensors (e.g., color cameras) commonly used with mobile robots. Finally, the SPHERES do not have a 
high-bandwidth wireless communications link, which would support standard IP-based data messaging. 

To remedy this, we have added a commercial smartphone (Samsung Nexus-S) as an embedded controller to 
SPHERES. The smartphone upgrade provides SPHERES with a high-performance processor, significant on-board 
memory, color cameras, additional sensors (temperature, sound, gyroscopes, accelerometers), a touchscreen display, 
and Wi-Fi networking. The Nexus-S smartphone is based on the open-source Android operating system, which is 
highly flexible, scalable, and extensible. For clarity, we refer to the combined smartphone and SPHERES system as 
“Smart SPHERES” (Figure 2). 
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Figure 2. Smart SPHERES combines a modified Android smartphone with SPHERES. 

The smartphone is attached to SPHERES via the expansion port. This image shows the 
robot in a laboratory at NASA Ames, where it is tested using an air carriage on a granite table. 

One key advantage of using off-the-shelf consumer devices is that we can very rapidly research and develop new 
capabilities at a significantly lower cost than with traditional space flight hardware. In particular, modern smart- 
phones contain high-performance processors that exceed the capabilities of existing flight hardware at a fraction of 
the cost. The Nexus-S, for example, includes a 1 GHz ARM CPU, a high-performance GPU (graphical processing 
unit), 512 MB RAM, and 16 GB of non-volatile storage. When first released, this smartphone sold for approxi- 
mately $500. In comparison, the current RAD750 single-board computer, which is one of the most widely used rad- 
hard flight processors, operates at 200 MHz and costs more than $200,000 - just for the CPU. 

Moreover, with the rapid evolution of mobile communications device technology we can regularly upgrade the 
computing and sensing capabilities of Smart SPHERES. Smartphone processors are also extremely rugged, com- 
pact, and low-power. All of these characteristics make them ideal for use as SCADA (Supervisory Control and Data 
Acquisition) processors. In addition, future smartphones are expected to incorporate additional sensors, such as mul- 
tiple cameras, or 3D ranging devices, which are well suited for robotics applications. 

In order to certify the Smartphone for use on ISS, we had to make three modifications to the device. First, to 
avoid any possible radio frequency interference with ISS, we removed the cellular GSM transmitter chip from the 
device. Second, to address concerns about the use of lithium-ion batteries on ISS, we developed an external battery 
pack using AA alkaline batteries. Finally, to capture and contain the display glass in the event of breakage, we cov- 
ered the touchscreen with Teflon tape. 

The Smartphone is connected to SPHERES via a custom cable, which connects the Smartphone USB port (in 
RS-232 serial mode) to the SPHERES expansion port. The Smartphone sends motion commands to the SPHERES 
and receives low-level telemetry (power, position, etc.) messages across this serial link. 
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B. Robot User Interface 



Figure 3. The Smart SPHERES Workbench provides multiple modes for 
remotely commanding and monitoring the Smart SPHERES robot. The interface contains 
numerous displays including smartphone camera imagery and interactive 3D maps. 

Smart SPHERES is remotely operated using the Smart SPHERES Workbench (Figure 3). The Workbench con- 
sists of software services for executing robot commands (robot trajectories, sensor data collection, etc.), displaying 
telemetry, and monitoring robot activities with interactive 3D maps. The Workbench is implemented in Java using 
the Eclipse Rich Client Platform (RCP) framework and provides multiple modes for commanding and monitoring 
Smart SPHERES operations. 

At present, the Workbench supports two control modes. The primary mode, “Run Plans”, enables the robot to be 
operated in a supervisory control manner 9 . With “Run Plans”, the operator selects a plan that contains a sequence of 
commands (fly to point, orient to pose, acquire sensor data, etc), which is then transmitted to the robot for autono- 
mous execution. The operator then monitors the robot, intervening only when necessary (e.g., when autonomy has 
difficulty or fails). This control method is inherently robust to communications latency and intermittent loss-of- 
signal, as well as ensures consistent task performance (i.e., does not suffer from the effects of variable operator pro- 
ficiency, workload, or fatigue). 

The secondary mode, “Manual Control”, enables the robot to be directly teleoperated. In this mode, the operator 
is directly in the control loop and issues discrete commands (i.e., incremental motions for forward, back, left, right, 
yaw, and hold) to the robot. To facilitate operator situation awareness, the Workbench provides multiple displays 
including a forward-facing camera view and 3D graphical views, which show robot position, camera field-of-view, 
etc. We consider “Manual Control” to be the fallback control mode because the efficacy of direct teleoperation is 
highly constrained by the quality of the communications link. Specifically, in situations where the link has delay, 
jitter, and/or low-bandwidth, remote tasks can only be performed in a slow manner. 

C. Data Communications 

The data communications architecture for Smart SPHERES is shown in Figure 4. A ground controller in the 1SS 
Mission Control Center (MCC) at the NASA Johnson Space Center, Houston, Texas uses the Smart SPHERES 
Workbench to execute commands for the robot. The commands are uplinked, and robot telemetry is downlinked, via 
the Orbital Communications Adaptor (OCA) LAN and the Tracking and Data Relay Satellite System (TDRSS) Ku- 
band link to the ISS. On-board the ISS, the robot data is routed to/from the Smart SPHERES via the Operations Lo- 
cal Area Network (OPS LAN) and the Joint Station Local Area Network (JSL) Wi-Fi. 
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Smart SPHERES uses a combination of the NASA Robot Application Programming Interface Delegate (RAPID) 
framework, the Data Distribution System (DDS), and Delay Tolerant Networking (DTN) for data messaging. 
RAPID is an open-source framework for remote robot operations that is designed to: facilitate integration of ex- 
perimental robot software modules created by a distributed development team; improve the compatibility and reus- 
ability of robotic functions; and speed prototype robot development in a wide range of configurations and environ- 
ments 10 . RAPID supports robot commanding (primitive actions and command sequences), monitoring (telemetry 
including robot state, position, task progress, etc.), and transfer of large-volume datasets (e.g., large image sets). 

One significant benefit of RAPID is that it encourages the development of loosely coupled, but highly-cohesive 
systems across distributed and networked computing. This means that software modules are inherently portable and 
can be easily redeployed to fit different applications. Consequently, robot user interfaces, such as the SPHERES 
Workbench, can be developed for use at mission control, but then reconfigured for use on-board 1SS with relatively 
little effort 5 . 

In order to ensure that Smart SPHERES commands and telemetry are delivered reliably over multi-hop, delayed 
and disruption-prone communication links, we use both DDS and DTN 2 . DDS is an open, international middleware 
standard directly addressing publish-subscribe communications for real-time and embedded systems. DTN makes 
use of “store and forward” techniques within a data network to compensate for intermittent communication link 
availability or connectivity. 

D. Software Architecture 

Smart SPHERES uses a service-oriented software architecture, which is inspired by the NASA Ames Service 
Oriented Robotics Architecture (SORA) 4 . The architecture encapsulates robot functions as a collection of modular 
services, which can be assembled and configured for different applications using high-performance middleware. 

High-level services reside on the smartphone. The Motion and Action Sequencer is a simple task executive, 
which manages command sequence execution. The Camera service captures images from the Smartphone color 
camera. The On-Screen UI manages state and diagnostics display on the Smartphone touch-screen. The WiFi 
Comms service is a Robot Application Programming Interface Delegate (RAPID) middleware client 10 , which re- 
ceives commands and transmits telemetry. The SPHERES Comms service manages serial communication with the 
SPHERES. 

Low-level services reside on the SPHERES. The Mobility Control service provides 6-DOF motion control. The 
State Server service manages serial communications with the smartphone, which includes processing mobility (tra- 
jectory) commands and transmitting state (power, position, etc.) data. 
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V. ISS Deployment and First Test 


A. Test Overview 

We began development of the Smart SPHERES in November 2010 and deployed two modified Nexus-S smart- 
phones to the ISS in July 2011 on the last Space Shuttle flight (STS-135, Atlantis). ISS Expedition 29 Commander 
Mike Fossum subsequently unpacked and prepared the smartphones for in-space flight testing. 

The first on-orbit test of Smart SPHERES took place on November 1, 2011. During this test (Figure 5), Mike 
Fossum mounted the smartphone on a SPHERES, then commanded the SPHERES to fly a pre-defmed trajectory in 
the Kibo Japanese Experiment Module (JEM). During this initial test, we recorded data from numerous smartphone 
sensors, including the front color camera, accelerometers, gyroscope, and magnetometer. 



Figure 5. Astronaut Mike Fossum testing Smart SPHERES on the ISS. 

The Smartphone is shown in the inset (left). (NASA ISS029-E-036493 / -036497) 


We defined a JEM coordinate system (Figure 6, left) with X forward, Y starboard, Z toward the deck, and the 
origin in the center of the module. At the start of the test, the Smart SPHERES was placed at the origin and then was 
commanded to translate in sequence: 1 m forward (+X), back to center, 1 m starboard (+ Y), back to center, and lm 
toward deck (+Z), and back to center. Then robot then made a full rotation about each of the coordinate axes. 

After the on-orbit test, we ran a similar test in our laboratory at NASA Ames. On the ground, the robot floats in 
an air carriage on a granite table (Figure 2) and is constrained to three degrees of freedom. The coordinate system 
for the ground laboratory is shown in Figure 6 (right). 



Figure 6. Coordinate systems for Smart SPHERES. 

Left: Japanese Experiment Module (JEM), right: ground laboratory at NASA Ames. 
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B. Results 

Figure 7 shows data from the smartphone’s three-axis gyroscope that was collected while the Smart SPHERES 
performed a full rotation about its Y axis. In both plots, the horizontal axis shows time in nanoseconds and the verti- 
cal axis shows angular speed in radians per second. As the plots show, the robot moves quite differently in space 
than on the ground. The bottom plot shows rotation about Y, the top plot shows rotation about negative Y, so the 
robot were in fact rotating opposite directions during these tests. The plots each have four “humps” because of the 
way the motion was programmed; the robot rotated 90 degrees and paused, then another 90 degrees and paused, etc. 
The robot came closer to a complete stop between turns in orbit than it did on the ground. 

In space, the Smart SPHERES turned faster (reaching nearly -0.3 radians per second) than on the ground (barely 
0.2 radians per second). This difference is due to the ground unit riding on an air carriage, which adds additional 
mass to be moved along with small, though non-negligible friction. The firing of the cold gas thrusters can be seen 
in both plots, though the orbit plot (Figure 7, top) more clearly shows the “spikes” in angular rate associated with 
each firing. 

The blue ( X) and red (Z) traces in the ground plot (Figure 7, bottom) are both perfectly flat. This is because the 
robot is constrained by the air carriage, i.e., it is unable to rotate on those axes. In the orbit plot, however, the data 
shows a bobble in both X and Z as the robot aligned itself with the coordinate axes. The robot is well-aligned by the 
end of the first quarter rotation, but the X and Z traces show residual oscillation, which is indicative of motion con- 
trol performance limitations. 


Orbit - Rotation about JEM Z (Phone -Y) 



Ground - Rotation about Lab X (Phone Y) 



Figure 7. Gyroscope data for a pure single axis rotation. 
Top: on-orbit test, bottom: ground laboratory test. 


Figure 8 shows magnetometer data from the smartphone that gathered during the same tests. In both plots, the 
horizontal axis shows time in nanoseconds and the vertical axis shows magnetic field strength in microTesla. The 
plots show that the robot’s rotation on the ground is more stable than in orbit. Moreover, the plots suggest that mag- 
netic field strength in orbit is lower than on the ground. However, because we did not calibrate the magnetometer 
on-orbit (due to limited crew availability), additional testing will be needed to provide confirmation. 


American Institute of Aeronautics and Astronautics 


Orbit - Rotation about JEMZ (Phone -Y) 



Ground - Rotation about Lab X (Phone Y) 



The smartphone accelerometer was not sensitive enough to detect robot movement. Because SPHERES has a 
4 kg mass, but only 0.1 N thrusters, acceleration was within sensor noise (Figure 9). This is not surprising given that 
smartphone accelerometers are designed to detect gross motions rather than making high-precision measurements. 


Orbit 



Ground 



Figure 9. The Smartphone's linear accelerometer did not have sufficient sensitivity to detect robot motion. 
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VI. IVA Survey Test 


A. Test Overview 

On December 12, 2012, we used Smart SPHERES to carry out IVA surveys of the 1SS Kibo laboratory, using 
one of the color cameras built-in to the smartphone to record imagery (Figure 10). This test was designed to demon- 
strate that ground control can perform IVA tasks that require mobile sensors and that would normally be performed 
by astronauts. For example, ISS routine maintenance requires repeat surveys of interior environmental conditions 
(sound levels, temperature, etc.) to assess safety for habitation. Similarly, ISS logistic management requires repeat 
surveys for inventory and video documentation. These surveys, however, consume significant amounts of valuable 
crew time and could be performed by a ground-controlled robot. 



Figure 10. Smart SPHERES performing a video survey in the ISS Kibo laboratory 


The “IVA Survey Test” had three primary objectives: 

1) Demonstrate that a ground control operator can perform routine data gathering tasks with an IVA free- 
flyer that would normally require crew time 

2) Assess infusion readiness and technology gaps for routine use of IVA robotic free-flyer for surveys under 
ground control 

3) Mature technology required for ground controlled IVA free-flyer mobile sensors in the presence of com- 
munication time delays 

During the test, a typical activity involved the ground control operator selecting a survey plan with the Smart 
SPHERES Workbench, previewing the plan, and then uploading it to the Smart SPHERES for execution. As the 
robot carried out the plan, the operator monitored progress and paused/modified execution to handle contingencies. 
Some of the survey plans involved flying transects as command sequences. Other surveys required a combination of 
command sequences and manual control. Figure 11 illustrates the types of survey transects performed by Smart 
SPHERES. 
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FWD near Plan 


FWD near vertical Plan 



Figure 11. IVA Survey Plans 


At the start of the test, ISS Expedition 34 Commander Kevin Ford deployed the Smart SPHERES system in the 
ISS Kibo laboratory. He also affixed survey targets (based on the 1951 USAF 3 -bar test chart) to the Kibo walls and 
setup two video cameras, which provided live feeds for monitoring the test. These video feeds were not used by the 
ground control operator. After checkout, control of Smart SPHERES was transferred to ISS mission control. We 
performed ground control operation of Smart SPHERES from the “PLUTO” (PLug and port UTilization Officer) 
Multi-Purpose Support Room (MPSR). From this location, the Smart SPHERES Operations (SS Ops) engineer 
(Figure 12) remotely operated the robot using the Smart SPHERES Workbench. 



Figure 12. Smart SPHERES Ops (SS Ops) robot operator at ISS Mission Control 


The timeline of the key activities that took place during the test is given in Table 1. The timeline reflects both astro- 
naut and ground controller actions based on observations from two fixed ISS cameras, flight voice loops, and logs of 
the operator interface. 
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Table 1. Timeline of key activities during the IVA Survey Test. 


GMT 

Activity 

15:00 

Crew began preparing for the IVA Survey Test. 

15:59 

Crew affixed the survey targets to the forward wall of the Kibo module 

16:07 

Crew mounted the smartphone onto SPHERES. 

16:07 

Ground controller started the Workbench and verified connectivity to SPHERES. 

16:24 

Crew received “go’ from PAYCOM to start test program, which established communication be- 
tween the smartphone and SPHERES. This enabled ground control of Smart SPHERES. 

16:25 

Ground controller reported good connectivity to the robot. The Workbench displayed live smart- 
phone camera images and ground control sent a successful “echo” command to the robot. 

16:28 

Crew executed the SPHERES Quick Checkout program using the SPHERES facility GUI. This 
verified that the SPHERES was functional. 

16:31 

Ground control selected and executed “FWD near” plan. The robot successfully achieved every 
waypoint of the plan. The ground controller positively identified survey targets using the Work- 
bench image display. 

16:43 

Loss-of-signal (ISS out of TDRSS coverage) 

16:57 

Acquisition-of-signal (ISS back in TDRSS coverage) 

17:07 

Ground control selected and executed “FWD near vertical” plan. The robot hit the ceiling after 
waypoint #2, which caused localization to fail. The robot was then unable to reach waypoint #3, 
which the engineering team concluded was due to low propellant level. 

17:22 

Ground control selected and executed “FWD far” plan. The robot achieved the first waypoint, 
but then began moving sluggishly. Ground control halted the plan while the engineering team and 
crew discussed the status of the propellant. 

17:27 

Ground control re-ran “FWD far” plan. The robot again encountered problems after achieving 
the first waypoint. Ground control commanded Smart SPHERES to bypass waypoint 2 using 
manual control. The Smart SPHERES resumed execution of the plan. 

17:32 

Ground control paused the plan to further exercise manual control. Ground control executed all 
discrete commands (move up, down, left, right, forward, backward, yaw left, yaw right, hold) 
until loss-of-signal (ISS out of TDRSS coverage). 

17:37 

Acquisition-of-signal (ISS back in TDRSS coverage) 

17:37 

Ground control resumed the “FWD far” plan. The robot achieved waypoint #4, then proceeded to 
waypoint #5. Excessive downward momentum caused the robot to bounce off the floor mid- 
translation. The satellite then overshot the final waypoint, so ground control ended the plan. 

17:42 

Ground control selected and executed “FWD near” plan. 

17:46 

The robot began drifting. The engineering team concluded that the robot was out of propellant. 
Ground control ended the plan. 

17:47 

PAYCOM instructed crew to close out the test. 
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B. Results 

Our first objective for the IVA Survey Test was to demonstrate that a ground control operator can perform rou- 
tine data gathering tasks with an IVA free-flyer. Smart SPHERES successfully executed all commands sent from the 
ground control operator and all expected video and telemetry from the robot was received on the ground. The opera- 
tor attempted to carry out a total of five IVA survey plans. The first attempt was nominal from start to finish. An- 
other attempt resulted in the robot stalling at a waypoint. The plan was re-run, and the operator was able to manually 
bypass the problematic waypoint. Two of the plans ended due to issues with SPHERES propulsion, i.e., insufficient 
cold gas propellant for maneuvering. 

During the test, the ground controller was able to switch between supervisory control and manual control (direct 
teleoperation) modes. While in manual control, the Smart SPHERES successfully executed all discrete control 
commands. All of the manual control commands were performed while facing the forward wall of the Kibo labora- 
tory. Preliminary analysis shows that all of the commands were able to return the Smart SPHERES to its original 
pose. 

Our second objective was to assess infusion readiness and technology gaps for routine use of IVA robotic free- 
flyer. To assess the technical maturity of Smart SPHERES, we collected data to characterize robot motion, data 
communications, image quality, and system health. Table 2 lists the data that was continuously collected throughout 
the test session. 


Table 2. Data collected during IVA Survey Test. 


Characteristic 

Recorded data 

Position hold performance 

Commanded pose and actual pose 

Path tracking performance 

Commanded path and actual executed path 

Data communications 

Bandwidth consumed and message latency over time 

Survey coverage 

Commanded vs. actual plan (pose and camera pointing) 

Survey image quality 

Resolution, range to target, exposure (under/over) 

Operational display 

Workbench display of telemetry, health & status 
Robot log of telemetry, health & status 

Sensor data display 

Workbench display of sensor data 
Robot log of sensor data 


Preliminary analysis of the data indicates that the position hold performance of Smart SPHERES is adequate. 
During position hold with Smart SPHERES approximately 0.9 m from the forward wall, the camera successfully 
resolved line pairs of five mm based on the 1951 USAF resolution test chart. The smartphone automatic settings 
controlled the exposure of the frames well even when the overhead lights came into the frame. 

Smart SPHERES was not commanded to follow a path, the surveys were simply a series of position hold way- 
points. Our activity shows that this approach lacks the precision necessary to collect high-quality image data. Even 
though Smart SPHERES was commanded to reach zero velocity at a waypoint before moving to the next waypoint, 
the satellite retained enough momentum from the previous traverse to take it off a straight line between the way- 
points. 

In some cases, the camera did not capture all the desired image data because Smart SPHERES was not on the 
expected path. All surveys were designed to have a twenty percent overlap between image passes if executed per- 
fectly. The near survey, shown in Figure 13, had four passes at a distance of 70 cm from the wall. Because Smart 
SPHERES overshoot near the ceiling on this survey, it missed an area of about 500 square cm, or just over one per- 
cent of the 2 m square area being surveyed. The far survey had two passes at a distance of 1 10 cm from the wall; 
Smart SPHERES completed the majority of this survey and it was on track to miss 300 square cm, or slightly less 
than one percent of the target. In one case, overshoot caused Smart SPHERES to impact the ceiling with enough 
force to disrupt its position estimation and require that the survey be aborted. During the near test shown in Figure 
12, a gentle collision with the deck did not prevent Smart SPHERES from finishing the survey. 

Aside from difficulties traversing, another aspect of the Smart SPHERES motion system that poses challenges 
for image capture is undesirable rotation. The original SPHERES were symmetrical enough to be modeled as 
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spherical bodies in the control system. However, the addition of the expansion port and the smartphone made Smart 
SPHERES asymmetric. Our IVA surveys require that the camera face sideways to the direction of motion in order to 
image the wall. However, every time Smart SPHERES left a waypoint the camera would lag, pointing slightly 
backwards, and then swing forward quickly to catch up. As a result, the images taken near the start of a traverse 
were blurry, to the extent that no elements could be resolved on the 3 -bar test chart. Both imprecise path tracking 
and undesirable rotation could be mitigated by using a more precise motion control algorithm, and also by placing 
more waypoints along a desired straight line path. 

One unexpected difficulty in analyzing the data came from the number of protrusions from the 1SS walls. On the 
four square meters of the JEM forward wall that Smart SPHERES surveyed, a laptop, three handrails, a roll of Kap- 
ton tape, four hoses, and more than eight coils of wire jutted, or in some cases floated, out from the wall from 6 to 
more than 25 cm. These objects and their changing occlusions prevented the photostitching software we used from 
successfully stitching together the images from the near survey. The images from the far survey were able to be 
stitched much more successfully. The relative clutter of the ISS interior suggests that an IVA robot using stereo vi- 
sion would have sufficient texture for stereo correlation, but that a three dimensional model of the interior would be 
more appropriate than a two dimensional model. 

The Smart SPHERES operations engineer reported that the Workbench functioned well. In particular, the three- 
dimensional maps showing Smart SPHERES’s position estimate and the current survey path were much more help- 
ful in diagnosing robot behavior than the standard video feeds that are set up for each SPHERES session. 



Figure 13. Smart SPHERES path during one survey (shown with the commanded waypoints in plan order). 

Smart SPHERES reached each waypoint, but it did not follow a straight line between them. 

VII. Conclusion 

We developed Smart SPHERES to assess how a telerobotic free-flying robot can perform a variety of IVA tasks. 
To date, we have been studying how ground controllers can remotely operate Smart SPHERES on the ISS to per- 
form IVA surveys. The capacity to perform such surveys is fundamental to environmental monitoring, inventory, 
inspection, etc. Our on-orbit testing indicates that telerobotic, free-flying IVA surveys can be successfully performed 
using a combination of supervisory control and teleoperation. 

Specifically, when the Smart SPHERES was able to reach commanded waypoints, it provided acceptable im- 
agery that allowed the ground controller to identify specific features on the JEM forward wall inside the ISS. The 
telemetry and video downlinked to the Smart SPHERES Workbench also appeared to provide the ground controller 
with excellent situation awareness. The SmartSPHERES command and control architecture gave near full control of 
robot operations to the ground controller. 


14 

American Institute of Aeronautics and Astronautics 


However, several improvements need to be made in order to mature the Smart SPHERES from prototype into a 
fully-operational flight system. First, the existing robot is constrained in many ways by the original SPHERES de- 
sign. These include limited run-time, limited payload mass capacity, reliance on an external positioning system 
(which requires manual installation and calibration), lack of hazard sensors (for collision detection and avoidance), 
etc. Second, the Smart SPHERES needs the capacity to self-recharge and self-resupply, e.g., to replenish electrical 
power and propellant. This capacity is important to help minimize the time and effort required by astronauts to 
maintain the robot. Third, the Smart SPHERES needs to be able to detect, and interact with, astronauts it encounters 
- so that it will be perceived as integral to the IVA environment and not seen as an intruding nuisance. Finally, the 
Smart SPHERES needs to be engineered to be significantly more robust and reliable, so that it can be integrated into 
the ISS as a long-term core system. 
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